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ABSTRACT 

An autonomous intelligent training system 
which integrates expert system technology 
with training/teaching methodologies is de- 
scribed. The system was designed for use by 
Mission Control Center (MCC) Flight Dynamics 
Officers (FDOs) training to perform payload-as- 
sist module (PAM) deploys from the orbiter. 
The system (termed PD/ICAT for Payload-as- 
sist module Deploys/Intelligent Computer- 
Aided Training system) is composed of five 
distinct components: a user interface, a domain 
expert, a training session manager, a trainee 
model, and a training scenario generator. A 
user interface has been developed which per- 
mits the trainee to access data in the same 
format as it is presented on console displays in 
the MCC. The interface also permits the 
trainee to take actions in much the same man- 
ner as a FDO in the MCC and provides the 
trainee with information on the current train- 
ing environment and with on-line help (if 
permitted by the training session manager). 
The domain expert (DeplEx for Deploy Expert) 
contains the rules and procedural knowledge 
needed by a FDO to carry out a PAM deploy. 
DeplEx also contains "mal-rules" which permit 
the identification and diagnosis of common er- 
rors made by the trainee. The training session 
manager (TSM) examines the actions of the 
trainee and compares them with the actions of 
DeplEx in order to determine appropriate re- 
sponses. A unique feature of the TSM is its 
ability to grant the trainee the freedom to fol- 
low any valid path between two stages of the 
deploy process. A trainee model is developed 
for each individual using the system. The 
model includes a history of the trainee’s inter- 
actions with the training system and provide^ 


evaluative data on the trainee’s current skill 
level. Following each trainee action, evalua- 
tive assertions are made by the TSM and used 
to update the trainee model. A training sce- 
nario generator designs appropriate training 
exercises for each trainee based on the trainee 
model and the training goals. PD/ICAT is cur- 
rently being tested by both experienced and 
novice FDOs in order to refine the system and 
determine its efficacy as a training tool. Ulti- 
mately, this project will provide a vehicle for 
developing a general architecture for intelli- 
gent training systems together with a software 
environment for creating such systems. 

INTRODUCTION 

The Mission Operations Directorate (MOD) at 
NASA/Johnson Space Center is responsible for 
the ground control of all space shuttle opera- 
tions. Those operations which involve alter- 
ations in the shuttle’s orbit characteristics are 
guided by a FDO who sits at a console in the 
"front room" of the MCC. Currently, the train- 
ing of the FDOs (called "fidos") in flight opera- 
tions is carried out principally through the 
study of flight rules, training manuals, and 
"on-the-job training" (OJT) in integrated sim- 
ulations. From two to four years is normally 
required for a trainee FDO to be certified for 
many of the tasks for which he is responsible 
during shuttle missions. OJT is highly labor 
intensive and presupposes the availability of 
experienced personnel with both the time and 
ability to train novices. As the number of ex- 
perienced FDOs has been reduced through re- 
tirement, transfer (especially of Air Force per- 
sonnel), and promotion and as the preparation 
for and actual control of missions occupies 
most of the MCC’s available schedule, OJT has 
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become increasingly difficult to deliver to 
novice FDOs. As a supplement to the existing 
modes of training, the Orbit Design Section 
(ODS) of the MOD has requested that the Arti- 
ficial Intelligence Section (AIS) of the Mission 
Support Directorate assist in developing an 
autonomous intelligent computer-aided train- 
ing system. After extensive consultation with 
ODS personnel, a particular task was chosen to 
serve as a proof of concept: the deployment of 

a PAM satellite from the shuttle. This task is 
complex, mission-critical and requires skills 
used by the experienced FDO in performing 
many of the other operations which are his re- 
sponsibility. 

The training system is designed to aid novice 
FDOs in acquiring the experience necessary to 
carry out a PAM deploy in an integrated sim- 
ulation. It is intended to permit extensive 
practice with both nominal deploy exercises 
and others containing typical problems. After 
successfully completing training exercises 
which contain the most difficult problems, to- 
gether with realistic time constraints and dis- 
tractions, the trainee should be able to suc- 
cessfully complete an integrated simulation of 
a PAM deploy without aid from an experi- 
enced FDO. The philosophy of the PD/ICAT 
system is to emulate, to the extent possible, 
the behavior of an experienced FDO devoting 
his full time and attention to the training of a 
novice--proposing challenging training sce- 
narios, monitoring and evaluating the actions 
of the trainee, providing meaningful comments 
in response to trainee errors, responding to 
trainee requests for information and hints (if 
appropriate), and remembering the strengths 
and weaknesses displayed by the trainee so 
that appropriate future exercises can be de- 
signed. 

BACKGROUND 

Since the 1970's a number of academic and 
industrial researchers have explored the ap- 
plication of artificial intelligence concepts to 
the task of teaching a variety of subjects (e.g., 
geometry, computer programming, medical 
diagnosis, and electronic troubleshooting). A 
body of literature is now extant on student 
models and teaching/tutoring methodologies 
adapted to intelligent tutoring systems in the 
academic environment 1 . The earliest pub- 
lished reports which suggested the applica- 


tions of artificial intelligence concepts to 
teaching tasks appeared in the early 1970's. 2 * 3 
Hartley and Sleeman 3 actually proposed an ar- 
chitecture for an intelligent tutoring system. 
However, it is interesting to note that, in the 
fourteen years which have passed since the 
appearance of the Hartley and Sleeman pro- 
posal, no agreement has been reached among 
researchers on a general architecture for in- 
telligent tutoring systems 4 . Nonetheless, a 
study of the literature on intelligent tutoring 
systems is an essential starting point for the 
development of the elements of an intelligent 
training system. 

Among the more notable intelligent tutoring 
systems reported to date are SOPHIE 5 , 
PROUST 6 and the LISP Tutor 2 . The first of 
these systems, SOPHIE, was developed in re- 
sponse to a U.S. Air Force interest in a com- 
puter-based training course in electronic trou- 
bleshooting. SOPHIE contains three major 
components: an electronics expert with a gen- 

eral knowledge of electronic circuits, together 
with detailed knowledge about a particular 
type of circuit (in SOPHIE this was an IP-28 
regulated power supply); a coach which ex- 
amines student inputs and decides if it is ap- 
propriate to stop the student and offer advice; 
and a troubleshooting expert that uses the 
electronics expert to determine which possible 
measurements are most useful in a particular 
context. Three versions of SOPHIE were pro- 
duced and used for a time but none was ever 
viewed as a "finished" product. One of the 
major lacks of the SOPHIE systems was a user 
model. It is interesting to note that the devel- 
opment of a natural language interface for SO- 
PHIE represented a large portion of the total 
task. 

PROUST and the LISP Tutor are two well- 
known intelligent tutoring systems that have 
left the laboratory and found wider applica- 
tions. PROUST (and its offspring, Micro- 
PROUST) serves as a "debugger" for finding 
nonsyntactical errors in Pascal programs 
written by student programmers. The devel- 
opers of PROUST claim that it is capable of 
finding all of the bugs in at least seventy per- 
cent of the "moderately complex" program- 
ming assignments that its examines. PROUST 
contains an expert Pascal programmer that can 
write "good" programs for the assignments 
given to students. Bugs are found by matching 
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the expert’s program with that of the student; 
mismatches are identified as "bugs" in the stu- 
dent program. This ability is contained • in the 
PROUST "bug rule” component. After finding a 
bug, PROUST provides an English-language de- 
scription of the bug to the student, enabling 
the student to correct his error. The system 
cannot handle student programs that depart 
radically from the programming "style" of the 
expert. The LISP Tutor is currently used to 
teach the introductory Lisp course offered at 
Carnegie-Mellon University. This system is 
based on the ACT (historically, Adaptive Con- 
trol of Thought) theory and consists of four el- 
ements: a structured editor which serves as 

an interface to the system for students, an ex- 
pert Lisp programmer that provides an "ideal" 
solution to a programming problem, a bug cat- 
alog that contains errors made by novice pro- 
grammers, and a tutoring component that pro- 
vides both immediate feedback and guidance 
to the student. Evaluations of the LISP Tutor 
show that it can achieve results similar to 
those obtained by human tutors. One of its 

primary features is its enforcement of what its 
authors regard as a "good” programming style. 

TRAINING VERSUS TUTORING 

The PD/ICAT system was developed with a 
clear understanding that training is not the 
same as teaching or tutoring 8 . The NASA 

training environment differs in many ways 
from an academic teaching environment. 
These differences are important in the design 
of an architecture for an intelligent training 
system: 

a. Assigned tasks are often mission- 
critical, placing the responsibility for 
lives and property in the hands of 
those who have been trained. 

b. Personnel already have significant 
academic and practical experience to 
bring to bear on their assigned task. 

c. Trainees make use of a wide variety 
of training techniques, ranging from 
the study of comprehensive training 
manuals to simulations to actual on- 
the-job training under the supervi- 
sion of more experienced personnel. 

d. Many of the tasks offer considerable 
freedom in the exact manner in which 
they may be accomplished. 


FDO trainees are well aware of the importance 
of their job and the probable consequences of 
failure. While students are often motivated by 
the fear of receiving a low grade, FDO trainees 
know that human lives, a billion dollar shuttle, 
and a $100+ million satellite depend on their 
skill in performing assigned tasks. This means 
that trainees are highly motivated, but it also 
imposes on the trainer the responsibility for 
the accuracy of the training content (i.e., veri- 
fication of the domain expertise encoded in the 
system) and the ability of the trainer to cor- 
rectly evaluate trainee actions. PD/ICAT is 
intended, not to impart basic knowledge of 
mathematics and physics, but to aid the 
trainee in developing skills for which he al- 
ready has the basic or "theoretical" knowledge. 
In short, this training system is designed to 
help a trainee put into practice that which he 
already intellectually understands. The sys- 
tem must take into account the type of train- 
ing that both precedes and follows— building 
on the knowledge gained from training manu- 
als and rule books while preparing the trainee 
for and complementing the on-the-job training 
which will follow. Perhaps most critical of all, 
trainees must be allowed to carry out an as- 
signed task by any valid means. Such flexibil- 
ity is essential so that trainees are able to re- 
tain and even hone an independence of 
thought and develop confidence in their ability 
to respond to problems, even problems which 
they have never encountered and which their 
trainers never anticipated. 

SYSTEM DESIGN 

The PD/ICAT system is modular and consists 
of five basic components: 

1. A user interface that permits the 
trainee to access the same informa- 
tion available to him in the MCC and 
serves as a means for the trainee to 
assert actions and communicate with 
the intelligent training system 

2. A domain expert (DeplEx) which can 
carry out the deployment process 
using the same information that is 
available to the trainee and which 
also contains a list of "mal-rules" 
(explicitly identified errors that 
novice trainees commonly make). 

3. A training session manager TSM) 
which examines the assertions made 
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by the DeplEx (of both correct and in- 
correct actions in a particular context) 
and by the trainee. Evaluative asser- 
tions are made following each trainee 
action. In addition, guidance can be 
provided to the trainee if appropriate 
for his skill level. 

4. A trainee model which contains a 
history of the individual trainee's in- 
teractions with the system together 
with summary evaluative data. 

5. A training scenario generator that de- 
signs increasingly-complex training 
exercises based on the current skill 
level contained in the trainee’s model 
and on any weaknesses or deficien- 
cies that the trainee has exhibited in 
previous interactions. 

Figure 1 contains a schematic diagram of the 



FIGURE 1 - PD/ICAT ARCHITECTURE 


PD/ICAT system. Note that provision is made 
for the user to interact with the system in two 
distinct ways and that a supervisor may also 
query the system for evaluative data on each 
trainee. The blackboard serves as a common 
"factbase" for all five system components. 
With the exception of the trainee model, each 
component makes assertions to the black- 
board, and the rule-based components look to 
the blackboard for facts against which to pat- 
tern match the left-hand sides of their rules. 


User Interface 

The primary factor influencing the interface 
design was fidelity to the task environment. 
To avoid negative training, it was deemed es- 
sential that the functionality and, to the extent 
possible, the actual appearance of the training 
environment duplicate that in which the task 
is performed. Figure 2 contains a view of the 



typical display seen by a trainee on a 

Symbolics 3600 series LISP machine. The 

upper right corner of the display contains 
menus that allow the trainee to make requests 
of other flight controllers, respond to requests 
from other controllers, call up displays, obtain 
information about the current or previous step 
in the deploy process, request help from the 

training system, and return to a previous step 
in the process. This menu has as many as 
three levels depending on the nature of the 

action taken by the trainee. Some actions are 
completely menu driven while others require 
the input of one or more "arguments”. All ac- 
tions taken by the trainee through these 
menus and the arguments that they may re- 
quire become assertions to the blackboard. All 
requests directed to the trainee and all mes- 
sages sent to the trainee in response to his re- 
quests or actions appear in a window in the 
upper left corner of the screen. These two 
portions of the screen serve to functionally 
represent the voice loop interactions that 
characterize the current FDO task environ- 
ment. Any displays requested by the trainee 
appear in the lower portion of the screen. 
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overlapped, if more than one is requested. 
Clicking the mouse on any exposed portion of a 
background display will bring it to the fore- 
ground. The displays replicate those seen by a 
FDO on console in the MCC. During develop- 
ment nominal data was supplied to these dis- 
plays (from a dedicated ephemeris-generating 
program or from "dummy" data sets) so that 
negative training would not occur. Experi- 
enced FDOs using PD/ICAT have expressed 
satisfaction with the user interface. 


(A) PREVIOUS EVENTS 
TRIGGER SECTION 
OF DEPLEX CODE 


(B) TRAINEE ACTION 
MATCHES OPTION 
ASSERTED BY 
DEPLEX 


BLACKBOARD 



DeplEx 

The Deploy Expert is a "traditional” expert 
system in that it contains if-then rules which 
access data describing the deploy environment 
and is capable of executing the PAM deploy 
process and arriving at the correct "answers". 
In addition to "knowing" the right way to con- 
duct the PAM deploy, DeplEx also contains 
knowledge of the typical errors that are made 
by novice FDOs. In this way, PD/ICAT can not 
only detect an erroneous action made by a 
trainee, but also, through these so-called "mal- 
rules", it can diagnose the nature of the error 
and provide an error message to the trainee 
specifically designed to inform the trainee 
about the exact error made and correct the 
misconception or lack of knowledge which led 
to the commission of that error. Another of 
the interesting features of the PD/ICAT system 
is its continual awareness of the environment 
(the external constraints dictated by the 
training exercise) and the context of the exer- 
cise. Rather than having DeplEx generate a 
complete and correct solution to the deploy- 
ment problem, only those actions which are 
germane to the current context are asserted. 
In this way the expert "adapts" to alternate, 
but correct, paths that the trainee might 
choose to follow. Figure 3 shows schematically 
how DeplEx operates. This strategy was 
adopted because the human experts that 
perform PAM deploys recognize that many 
steps in the deploy process may be 
accomplished by two or more equally valid 
sequences of actions. To grant freedom of 
choice to the FDO trainee and to encourage in- 
dependence on his part, the experts felt that it 
was essential to build this type of flexibility 
into the PD/ICAT system. 


(C) MATCHED OPTION 
REASSERTED AS 
LATEST EVENT 



(D) UNUSED OPTIONS 
DELETED BEFORE 
NEXTSTEP 


EVENT 1 
EVENT 2 
EVENT 3 
EVENT 4 


STEP 5 


FIGURE 3 - DeplEx Operation, Illustrating Us 
Adaptability to Valid, Usar-Salectsd Alternatives 


Training Session Manager 

The training session manager is dedicated 
principally to error-handling. Its rules com- 
pare the assertions of DeplEx with those of the 
trainee to detect errors. Subsequently, DeplEx 
asserts facts that allow the TSM to write ap- 
propriate error messages to the trainee 
through the user interface. In addition, TSM is 
sensitive to the skill level of the trainee as 
represented by the trainee model. As a result, 
the detail and "tone" of error messages is cho- 
sen to match the current trainee. For example, 
an error made by a first time user of the 
training system may require a verbose expla- 
nation so that the system can be certain the 
trainee will have all of the knowledge and 
concepts needed to proceed. On the other 
hand, an experienced trainee may have mo- 
mentarily forgotten a particular procedure or 
may have "lost his place". In this latter case a 
terse error message would be adequate to al- 
low the trainee to resume the exercise. The 
TSM also encodes all trainee actions, both cor- 
rect and incorrect, and passes them to the 
trainee model. 
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Trainee Model 

Successful intelligent tutors incorporate stu- 
dent models to aid in error diagnosis and to 
guide the student’s progress through the tu- 
tor's curriculum 9 . The trainee model in the 
PD/ICAT system stores assertions made by the 
TSM as a result of trainee actions. Thus, at its 
most fundamental level, the trainee model 
contains, for the current session, a complete 
record of the correct and incorrect actions 
taken by the trainee. At the conclusion of 
each training session, the model updates a 
training summary which contains information 
about the trainee's progress such as a skill 
level designator, number of sessions com- 
pleted, number of errors made (by error type 
and session), and the time taken to complete 
sessions. After completing a session, the 
trainee can obtain a report of that session 
which contains a comprehensive list of correct 
and incorrect actions together with an evalua- 
tive commentary. A supervisor can access 
each trainee’s model to obtain this same report 
or to obtain summary data, at a higher level, 
on the trainee's progress. Finally, the training 
scenario generator uses the trainee model to 
produce new training exercises. 

Training Scenario Generator 

The training scenario generator relies upon a 
database of task "problems" to structure 
unique exercises for a trainee each time he 
interacts with the system. The initial exercises 
provided to a new trainee are based on vari- 
ants of a purely nominal PAM deploy with no 
time constraints, distractions or "problems". 
Once the trainee has demonstrated an accept- 
able level of competence with the nominal de- 
ploy, the generator draws upon its database to 
insert selected problems into the training en- 
vironment (e.g., a propellant leak which ren- 
ders the thrusters used for the nominal sepa- 
ration maneuver inoperable and requires the 
FDO to utilize a more complicated process for 
computing the maneuver). In addition, time 

constraints are "tightened" as the trainee gains 
more experience and distractions, in the form 
of requests for information from other MCC 
personnel, are presented at "inconvenient" 
points during the task. The generator also ex- 
amines the trainee model for particular types 
of errors committed by the trainee in previous 


(and the current) sessions. The trainee is then 
given the opportunity to demonstrate that he 
will not make that error again. Ultimately, the 
trainee is presented with exercises which em- 
body the most difficult problems together with 
time constraints and distractions comparable 
to those encountered during integrated simu- 
lations or actual missions. 

SYSTEM INTEGRATION 

The PD/ICAT system is currently operational 
on a Symbolics 3600 series Lisp machine. The 
user interface and trainee model are written 
in common Lisp while the rules of DeplEx, 
TSM, and the training scenario generator are 
written in ART 3.0. The system will ultimately 
be delivered to MOD in a Unix workstation 
environment. To accomplish this delivery, the 
ART rules were written to facilitate translation 
into CLIPS 10 and the Lisp code will be 
converted into C. It is uncertain, until the 
exact delivery environment is specified, how 
well the user interface can be ported. 

CONCLUSIONS 

The PD/ICAT system has, so far, proven to be 
a potentially valuable addition to the training 
tools available for training Flight Dynamics Of- 
ficers in shuttle ground control. The authors 
are convinced that the basic structure of 
PD/ICAT can be extended to form a general 
architecture for intelligent training systems 
for training flight controllers and crew mem- 
bers in the performance of complex, mission- 
critical tasks. It may ultimately be effective in 
training personnel for a wide variety of tasks 
in governmental, academic, and industrial set- 
tings. 
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ACRONYMS 


AIS 

Artificial Intelligence Section 

DeplEx 

Deploy Expert 

FDO(s) 

Flight Dynamics Officer(s) 

MCC 

MOD 

Mission Control Center 
Mission Operations Directorate 

ODS 

OUT 

Orbit Design Section 
on-the-job training 

PAM 

PD/ICAT 

Payload-Assist Module 
Payload-Assist Module 
Deploy/Intelligent Computer-Aided 
Training 

TSM 

Training Session Manager 
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